Greptile Summary

This PR adds two industry-standard evaluation suites to benchmark Stagehand's web automation capabilities: WebVoyager (643 test cases) and GAIA (90 test cases). The changes significantly expand the evaluation infrastructure to support data-driven benchmarking against established datasets.

The core architectural change introduces a suite-based evaluation system alongside the existing task-based approach. New suite builders (evals/suites/webvoyager.ts and evals/suites/gaia.ts) read JSONL dataset files and dynamically generate test cases, while corresponding task implementations (evals/tasks/agent/webvoyager.ts and evals/tasks/agent/webarena_gaia.ts) execute the actual evaluations. The system supports flexible sampling strategies using Fisher-Yates shuffle for randomized selection or deterministic first-N selection.

Key infrastructure improvements include:

• A new core/summary.ts module that extracts summary generation logic into a reusable component
• Enhanced type system with optional taskParams and params fields to pass dataset-specific parameters to evaluation functions
• New utility functions for JSONL parsing, data validation, and sampling in evals/utils.ts
• Environment variable configuration for controlling test execution (sample sizes, limits, difficulty levels)
• Updated evaluation runner logic in index.eval.ts to handle both static tasks and dynamic dataset-driven evaluations

The datasets themselves are substantial additions: WebVoyager contains 643 web navigation tasks across 13+ websites (Amazon, Google services, GitHub, etc.), while GAIA provides 90 general AI assistant tasks with varying difficulty levels. Both datasets start from standardized URLs and expect structured response formats.

This integration maintains full backward compatibility with existing evaluations while providing the foundation for systematic benchmarking against industry standards. The sampling capabilities allow for both development testing (small samples) and comprehensive evaluation runs.

Confidence score: 4/5

• This PR is safe to merge with minimal risk as it maintains backward compatibility and adds well-structured evaluation capabilities
• Score reflects solid implementation patterns and comprehensive infrastructure changes, though there's a potential division-by-zero edge case in summary generation
• Pay close attention to evals/core/summary.ts for the division-by-zero issue in category success rate calculation

_{12 files reviewed, 3 comments}

_{Edit Code Review Bot Settings | Greptile}